%%This is a very basic article template.
%%There is just one section and two subsections.
\documentclass[11pt,oneside,a4paper]{article}
\usepackage{graphicx} 
\usepackage{cite}
\usepackage{pdfpages}
\usepackage{listings}
\usepackage{color}
%\usepackage[titletoc.title]{appendix}
%\usepachage[dotinlabels]{titletoc}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}

\lstset{frame=tb,
  language=Java,
  aboveskip=3mm,
  belowskip=3mm,
  showstringspaces=false,
  columns=flexible,
  basicstyle={\small\ttfamily},
  numbers=none,
  numberstyle=\tiny\color{gray},
  keywordstyle=\color{blue},
  commentstyle=\color{dkgreen},
  stringstyle=\color{mauve},
  breaklines=true,
  breakatwhitespace=true
  tabsize=3
}

\title{Emotion Recognition for Mobile Devices with a potential use in Serious Games for Autism Spectrum Disorder}
\author{Adiljan Abdurihim and Olav Brenna Hansen}

\begin{document}
\maketitle

\section{Introduction}\label{s:intro}
Humans get a lot of emotional information through visual communication. This is essential for our understanding and interpretation of emotions. It´s also most recently becoming a mainstream product with spin off from eg MIT \cite{Affectiva}. In the aspect of serious games for autism Helmi Adly Mohd Noor, Faaizah Shahbodin and Naim Che Pee gives a good review of related literature \cite{Review}. The most familiar impacts of ASD (autism spectrum disorder) is in social cognition, communication and imagination. Autism is a very well researched area within science and can be divided into five categories: autistic disorder, asperger's syndrome, rett's disorder, childhood disintegrative disorder and pervasive development disorder\cite{Review}. 
The frequency of the disorder in Norway is one in a thousand and growing \cite{Stat}. One important aspect is capability people with ASD is their difficulties to communicate in social life. Research indicates that individuals with autism show no clear deficits when matching basic expressions or detection subtle expressions when presented for a longer duration. However, they appear to have difficulties when they are processing simple expressions rapidly and discerning subtle differences in complex expressions \cite{weigelt2012face}.

\subsection{ADS case study}\label{ref:caseStudy}
Contribution to science is to provide a solution that can be useful to be used in various edutainment context and in this case the project develops a face recognition solution to provide possible enhancement of social and communication adaptations for users with autism \cite{weigelt2012face}. The solution will be used for edutainment by motivating and engaging them in a way to potentially better their understanding of emotions \cite{zeeland2010reward}. By attaching game mechanics in the development of a serious game, it will provide both entertainment and education of the different emotional states, sensus. Social cognition is one of important factors while dealing autism. The purpose of game design  is to enhance self-efficacy(section \ref{s:scenario}). Additionally, the solution enables the possibility to enlarge, extend, exchange the context of use. 

\subsection {The power of mobile devices}
Jesse Schell on Gamification. Designing outside the box\cite{JesseSchell}.

Always with me, app boom, powerful to provide techniques to develop desired solution.
Using mobile devices can also have an impact on cost, see section \ref {ref:Cost}.



\subsection {Game as a motivator}

Jane Magonigal on games making a better world\cite{mcgonigal2011reality}

\begin{itemize}
\renewcommand{\labelitemi}{$\bullet$}
\item Urgent optimism
\item Social fabric-- like people better (thrust)
\item Blissful productivity -- happier working hard
\item Epic meaning
\\
= super empowered hopeful individuals but virtual saving the world 
\end{itemize} 

Games attract people and motivate people to do a task, therefore we believe that functionality like smile recognition and etc can be a useful functionality to use it in Serious Gaming with thought of enhancement of communication and social life of people with %Autism.
\\
Game mechanics and similar solution that gave good results.
eyes focused on what’s happening, intense mental concentration, relishing the challenge, totally in control and looks of absolute delight when they achieve their goals. 


This is a learning nirvana.

\subsection {Effectiveness of Visual education in Autism}\label {ref:Cost}
The Fleming’s VARK model gives a representation of  a well used learning style. It appears that autistic individuals are more likely to rely on only one style of learning. It's benefitial so find out what kind of lerning style the person favours. If the person likes to watch video, looking at books with pictures or observing persons, the visual learning style might be right. For those using their hands a lot puseling with objects and moving things around, it may indicate a 'hands on' learner. The auditory learner tends to talk a lot and enjoy onversations and like to lissen to music. Most people with ASD tend to be visual learners.. Linda Hogden \cite{Quirk} regards thrue observations   that:
\\
\\
 "It is best to think of people with autism “as being 90\% visual and 10\% auditory learners".

 In Sintavilla there will mostly be a visual leaning style although some text will be attached explaining the storytelling using a auditory learning style for the following reasons:


\begin{itemize}
\renewcommand{\labelitemi}{$\bullet$}
\item Enjoy interacting with technology and video games\cite{Review}.
\item Visual skills of people with autism are often superior to their skills in other areas. 
\item People with autism claim to use visual information to interpret their world.
\item People with autism are better able to comprehend permanent (non-transient) visual information because the message is present long enough for them to take in and process the information.
\item Programs that use visual strategies are highly effective with people who have autism.


\end{itemize} 


\subsection{Relevant solutions}\label{ref:solutions}
In order to contribute to the development in serious games for health, focusing on autism, it's necessary to review other similar solutions. Teacch should be mentioned in this setting. It's not a single technique or method but a complete program of services for autistic people which makes use of several techniques, of several methods in various combination depending upon the individual person's needs and emerging capabilities\cite{Teacch}. The proposed solution in this solution should be seen as a potential tool within this process not as a complete solution on its own. Relevant solutions are cMotion\cite{finkelstein2009cmotion}, Let's Face It\cite{tanaka2010using}, SmileMaze\cite {cockburn2008smilemaze} and an interactive toys \cite{barakova2007using} approach. They are all focused on autism and show an increasing use of new technology to interact with people with ASD. Some show good result and others need a deeper study to show its benefit, although they all reveal a need for good an innovative solutions in the area.  

There have been different approaches in making serious games for children with ASD. 
The main thing for them all seem to be the understanding that there is a need to learn them communication or social skills. Teacch should be mentioned in this setting. It's not a single technique or method but a complete program of services for autistic people which makes use of several techniques, of several methods in various combination depending upon the individual person's needs and emerging capabilities\cite{Teacch}. The main goal of Teacch for autistic children is to help them grow up to a maximum autonomy at adult age. This includes helping them understand the world that surround them, acquiring communication skills that will enable them to relate to other people and giving them as much as possible the necessary competence to be able to make choices concerning their own lives\cite{Teacch}.

\begin{itemize}

\renewcommand{\labelitemi}{$\bullet$}

\item By giving the person means to understand better his/her environment, it (the environment) becomes more predictable and less anxiety generator. This may require proposing a simpler environment in the early phases of development and progressively reintroducing complexity as the child progresses towards more and more autonomy.

\item By giving means of communication to the person the comprehension and expression capabilities will enable him/her to understand better what is being told/asked and to express his/her needs and feelings by other means than behaviour problems.

\end{itemize} 

\subsection{cMotion}
Uses Virtual humans and programming logic to teach emotion recognition to children\cite{finkelstein2009cmotion}.
By introducing avatars, cMotion hopes to increase users gaming motivation and therby their ability to learn. 

\subsection{Interactive toys}
Emilia Barakova et.al \cite{barakova2007using} uses a interactive toys in order to motivate and make children with ASD more aware. The children also showed a exploitative behaviour in using the toys. They targeted 5 to 9 years old in a controlled environment disconnected from strangers, though ASD children tend to become inhibited when interacting with strangers. They test the children’s interest for pattern and regularity.  

\subsection {Let's face it}
The program is a multimedia, computer-based intervention that is designed to teach face processing skills to children with autism. 
The aim is to develop effective treatments that will enhance the face processing skills of children with autism spectrum disorders (ASD) as well as other populations that exhibit impairments in facial processing \cite{tanaka2010using}.

\subsection{SmileMaze}
The goal of SmileMaze is to improve the expression production skills of children with
ASD in a dynamic and engaging game format\cite {cockburn2008smilemaze}. CERT automatically detects
frontal faces in a standard web-cam video stream\cite{littlewort2006dynamics}. SmileMaze integrates the use of
facial expression production into an intervention program
aimed at improving the facial expression recognition and
production skills of children with ASD. SmileMaze is intended to become a part of the Let's Face It program. 


\subsection{Serious game classification}
In this project an offline solution for an Android OS device with touch screen and integrated camera will be implemented. Even though the game solution is targeting health/therapy, in the way of improving ASD patients social skills and thereby how they communicate to others. There will not be made any statement that the game actually improves patient health. A edutainment classification \cite{Review}might be more appropriate until a further study on the games health improvements and how it correlate to people with ASD. 
\cite{charsky2010edutainment} :
\\
\\
"What is needed is a framework for designing serious games that builds off of the
edutainment and instructional games research. Although many edutainment and
instructional computer games are lacking in sophistication, depth of learning, and
game play, they offer a foundation for analysing why and how games enable
learning which can inform the design of serious games."
\\
\\
Furthermore the solution can be used in various context. Therefore people with autism will be used as a case-study to narrow the area of development. The underlying technology and research area can lead to different development directions. There are recognition of the whole body or part of the specific regions of the body. This project will focus on emotional recognition from the human face. Besides the latter a good development strategy would be to look at regular patterns, heuristic evolution and user centred process.       

\section{Sintavillie}


The first step for this project will be to make a face recognition system using pictures where the user gives a feedback on the actual emotion from the facial posture trying to recognize positive or negative valence\cite{krathwohl2002revision}. The term valence is used to categorize positive and negative emotions in the field of psychology. 

The second step includes android faceRecognition API and opencv \cite{OpenCV}. This give the capability for the user to take pictures using hardware in a mobile device from a given state of human emotion. Autism have many stages of illness and it would be to much of a challenge to implement a solution to cover them all. 

The game will focus on motivating patients with moderate autism with a expectation to potentially better their social interaction. The game benefits have to be studied in order to state its characteristics of a serious game for health (therapy) or a serious game for education (communication skills)\cite{weigelt2012face}.  

\subsection{Game mechanics}\label{ref:mechanics}
There are different game mechanics implemented in this game. The skinner box \cite{SkinnerBox}is used by awarding players with points when they have collected enough smiles and awarding them when they meet a certain threshold (new level, appendix \ref{ap:b}). The threshold represents another game mechanic: Achievements \cite{Achivements}.
\\
\includegraphics[width=40mm]{test5.png}
\includegraphics[width=40mm]{test6.png}
 As the player reaches a threshold, a achievement is gained. These mechanics are implemented in order to make the player follow the pre-set path of the game and in order to make the player play longer. Wikipedia links throughout the game makes use of tangential learning \cite{TangentialLearning}. The target group is people with ASD and thereby the wikipedia input is asserted to their educational needs. 


\subsection{Game design}
As mentioned a regular pattern in game play is advisable when motivating people with ASD. The storytelling, level progress and game mechanics will be affected “to keep it simple” for the user.

\begin{figure}[ht]
\centering
\includegraphics[width=5in]{Diagram.png}
\caption{{\small Relevant solutions}} \label{fig:compare}
\end{figure} 

\begin{figure}[ht]
\centering

\includegraphics[width=40mm]{test3.png}
\includegraphics[width=40mm]{test4.png}\\


\end{figure} 

Humans have a huge number of emotional expressions but to follow our stated development rules, only smiling and not smiling will be recorded for the game play. Introduction to play where user takes picture of family, with short description of different types of face beforehand. Therefore, there will be provided some picture samples that will illustrate different emotions, at this point we are going to focus on two types; happy, angry. User will be provided with extra information how to play before initiation of camera (to place face inside the rectangle). After completion of tutorial user will be able to start the game. 
Based on the adventure genre representing a sustain structure\cite{Simen} to align with the learning goal:
\begin{itemize}
\renewcommand{\labelitemi}{$\bullet$}
\item Background story - the game universe
\item Task description - what you need to solve
\item Mind game, puzzle
\item Assessment - a type of test
\item Debriefing - a summery of achievements

\end{itemize} 

\subsection{Scenario}\label{s:scenario}

The game will be built on history where a hero, user, will protect the island called “Sintavillie". The objective of the game is to collect as much as possible happy faces, here actually it will be focused on emotions. The “Hero” uses emotions and faces to make Sintavillie happier and peaceful. In some levels there will be need of collecting angry faces beside general happy faces see appendix \ref{ap:b}. The angry faces can be used to mask hero in the enemies castle to get near to the King Sint who try to spread the sadness and angry people in the Sintavillie. The pointing system of the game will use Skinner box idea, where user will gain points if he succeeds and lose points if wrong faces have been collected. At the end of game as at the start of each level and game there will be comics that will describe the process of game. The game will have a linear story narrative with the user in 1 person perspective. Different levels will help user with self-efficacy that will  affect human endeavour \cite{conner2005predicting}. 



\subsection{Benefits from new technology}
In a study from 2010, Kurt Squire \cite{squire2010information} shows in an educational game how new technology can be used to extend the classroom and prepare students for future participation in the real world. This is what people with ASD struggle with on a daily basis. Squire talks about an approach where new technology used by students can contributes to the research knowledge base. New technology can not only make this contribution more efficient but also give researchers instant access to data \cite{miller2012smartphone}\cite{chen2011service}. The data is essential in providing useful and effective feedback back to the users The huge amount of information online also gives a great posibility for up to date tangential learning (see section \ref{ref:mechanics}).

\subsection {Limitations}
The solution can be installed on off the shelf Android phones, SDK api level 8 and android version 2,2  or newer. There is no need for, or is the current version, suited for online usage at this stage of development. An online extension will increase the scientific contribution as well as the potential social interaction to other players. Although similar solutions (see section \ref{ref:solutions}) show good result in motivating and heightened awareness,  a study is needed to confirm the hypothesis for this solution. This will implicate a need to perform testing on people with ASD. 
\\
Another social aspect with the game, is the possibility to interact outside the classroom or home. By interacting with strangers, people with ASD might learn how to become less inhibited. 

\subsection {Cost}
There is always a cost involved in any development and serious games are usally cost more developing than games in general. The price will depend a lot on what technology, scope, variation and game genre \cite {Simen}.  A universal question is cost vs result. What will the overall cost be in developing the final result and will the benefits outweigh the increased health effect compared to more traditional methods. A draft of the economical costs are outlined in appendix \ref{ap:c}.

\section {Results}

Some people could gain knowledge from the device while other find it as a game for entertainment. The goal is to converge the functionality to adapt to the category or serious games and focusing on people with difficulties to interpret facial expressions. For this we focus on people with Autism but it can be used for other similar cases.
\\
Mobile devices are constantly adopting new technology and sensors \cite{lane2010survey} whereby  this technology can be implemented more easily into an future extended solution.

Using the mobile camera to interact and read other peoples emotions most highly increases the social skills for people with ASD and mostly the variety of contexts it can be used in, enhances the social experience compared to other only computer based solutions. 
\\
\\
The implementation of the functionality to the mobile device was done through use of OpenCV library and Android API. In order to get better understanding and retrieve results the implementation was done first on face part. Later on every part has been separated that allowed application to detect face, eyes, mouth. Finally, the smile recognition has been implemented. The incremental way of implementing the functionality build efficient code that can be used for different purposes. As emotions can be various there will also be a need to implement a posibillity to detect other emotions by detecting eyebrows, mouth etc. to detect angriness. Therefore the development approach used in this project can easily integrate new features (see section\ref{ref:caseStudy}). 
\\
The images below illustrate different tests done on random images and persons. These show that functionality available for recognition of smiles on images and real people.
\begin{figure}[ht]
\centering
\includegraphics[width=60mm]{test1.JPG}\includegraphics[width=60mm]{test2.JPG}\\
\includegraphics[width=60mm]{Lars.JPG}
\caption{{\small Recognizing facial expressions. The squares on the picture can be hidden and the white dot can be used to verify that the facial expression is found (in this case a smile).}} \label{fig:Emotion}
\end{figure} 

The images have rectangular lines that indicates different parts of face. The detected parts like mouth is indicated with a white dot that represents smile recognition. If dot is not there that means that there have not been found any smile on face. This kind of taking images is Indication of images in such way is just for better visualisation of the functioning recognition functionality and can be therefore removed if necessary. 

The beta version of the application is available in three levels (see appendix \ref{ap:b}).

\section{Conclusion}
What are possible future work for the people who will do the same work?


NOTE: Run bibtex build twice and after latex build to have correct output
below the bibliography is used for the setting up citation
\bibliographystyle{IEEEtran}
imports data from references.bib file
\bibliography{AutismDatabase}

\appendix{}
\section*{Appendicies}
This section includes all related files to the project and final report.



\section{Appendix}\label{ap:a}
\subsection *{OpenCV}
The official web page of OpenCV \cite{OpenCV} was the initial step  when setting up OpenCV in Eclipse IDE for the Android development. The documentation explained a step by step procedure for the integrating of the OpenCV library with the chosen IDE. Unfortunately, due to some changes in Mac environment, the NDK and make command for building the C/C++ code was failing. Therefore, the correct path with references had to be set manually in the .bash\_profile file. The second issue when setting up the environment was due to OpenCVs manager package flaw installing the emulator. This was solved by installing the appropriate manager package referenced in the tutorial. As the team furthered ahead it faced next difficulties. The deploying of apk was failing due to wrong reference in Android.mk file in JNI folder. Some other minor problems occurred due to different referencing of library.

\section{Appendix}\label{ap:b}
\subsection *{Plot}
Plot:
\\
Far in the north there is an island "Sintavillie" full of sadness. But long time ago it was a blooming island called Heldievillie before it was occupied by King Sint. Therefore, you must save people of Sintavillie collecting smile-faces and spread them in the island. Share smile, help people once again be able to smile and evolve happiness.

\subsubsection *{Levels}

Levels:
\\Level 1: Random of images with emotions: Map emotion, smile, from given images. if user finds the right image and choose it then he gets a text that tells that he did right otherwise there will be a text with a message that asks him to try again.
\\
\\Level 2: Collect smiling faces: Take picture of people around you, ask them to smile. If specified emotion is recognised user should take a picture. The OpenCV algorithm checks if the emotion is right one, if so it takes a picture and the user earns points, this will terminate camera and he will return to the level list with an ability move on further. The new level will appear. Otherwise he will still be at the same position and camera will indicate that he/she has to try again.
\\\\Level 3: Are you smiling? : Take picture of yourself to check if you are smiling. (same for being angry)
\\\\Level 4: My Family: Collect pictures that you will take to map individuals according to text. My daddy, my mom my brother…....granny... 
\\\\
Level 4a: My family smiling: Extra Images where everyone is smiling.
\\\\
Level 4b: My family angry: Everyone is angry.
\\\\Level 5: Recognize your family: Taken images in Level 3 and sub levels can be used for asking various questions like: Is it your daddy on image? Does your mom smile?.... etc
\\\\Level 6: Reversed or inverted?: Face inversion detection. 
\\\\Level 7: Avatar makes same emotion as me 
\\… .

\section{Appendix}\label{ap:c}
\subsection *{Cost}
Employees: 2 developers. Yearly cost 350.000.-
\\
Extra help: 2 professors. Cost per hour 1000.-
\\
\\
Including social expences:
\\
350.000.- x 1,32 = 462.000.-
\\
\\
Including material and working cost:
\\
462.000.- x 1,4 = 646.800.-
\\
\\
Total yearly cost developing = 1.293,600.-
\\
Estimate for 3 month developing = 323,400.-
\\
Ekstra help (consulting) = 12 hours x 2 professors x 1000.- = 24.000.-
\\
Total sum for project = 323,400 + 24,000.- = 347,400.- 



\section{Appendix}\label{ap:d}
Let's face it is based on an research based framework\cite{tanaka2003framework}.
Uses a domain model to describe face recognition and how people with ASD can train to get better. \\
Domain I: Attending to faces \\
DomainIIa: Facial identity\\
DomainIIb: Facial Expression\\
Domain III: Social meaning of facial cues\\
\\
Specific vategorization of perceptuell expertice \\
\\
New objects more easily recognized\\
New objects more easily learned\\
\\
Specific levels of categorization




\end{document}
